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I.  THE  CHANGING  INTELLIGENCE  PROBLEM  SPACE 


Since  the  fall  of  the  Soviet  Union,  socio-political  dynamics  in  the  World  have 
changed  dramatically.  While  larger  nations  in  the  World  cannot  forget  about  concerns 
about  and  readiness  for  traditional  nation-state  conflicts,  the  nature  of  what  have  come  to 
be  called  “irregular”  and  “asymmetric”  engagements,  coupled  to  significant  changes  in 
infonnation  technology  capabilities  have  resulted  in  an  intelligence  and  military  problem 
space  that  has  been  labeled  as  comprising  a  “mystery”  rather  than  a  puzzle  [1].  Similar 
problems  in  the  financial  world  gave  rise  to  the  notion  of  “Black  Swan”  events  involving 
surprise  and  high  impact  [2],  and  this  notion  has  carried  over  to  intelligence  and  military 
problems  as  well. 

Part  of  the  new  complexities  of  these  problems  is  that  they  occur  in  a  more  or  less 
“normal”  context  so  that  distinguishing  the  meaningful  from  the  irrelevant  involves 
knowing  about  both  settings  and  behaviors;  in  many  of  these  cases  the  “adversary”  is  not 
known.  Furthermore,  if  adversaries  are  known  they  exhibit  adaptive  behaviors  that  make 
understanding  and  predicting  threatening  or  anomalous  behaviors  very  difficult.  Our 
term  for  this  problem  class  is  “weak  knowledge”  problems  in  the  sense  that  reliable  a 
priori  models  or  procedural  knowledge  of  the  expected  adversarial  or  other  dynamics  are 
not  yet  available  and  thus  not  available  to  support  deductively-based  analyses.  As  a 
result,  analysis  approaches  to  such  problems  require  mixed  paradigms  where  any 
available  and  reliable  a  priori  knowledge  can  be  employed  in  a  deductive  framework  but 
at  the  same  time  inductive  and  abductive  methods  involving  learning  and  discovery 
operations  and  tools  are  also  part  of  any  modem  analytics  suite. 

Further  challenges  arise  in  these  cases  because  it  has  been  learned  that  inputs  well 
beyond  traditional  intelligence,  surveillance,  and  reconnaissance  (ISR)  sensor  data  are 
necessary  to  aid  in  inferencing  and  understanding  and  for  developing  situational 
hypotheses.  Intelligence  analysis  in  these  environments  requires  a  broad  range  of 
heterogeneous  data  and  information  to  include:  open  source  and  socio-political  data, 
contextual  infonnation,  ontological  information  (declarative  knowledge),  and  learned 
infonnation.  Much  of  this  data  is  of  a  type  that  is  today  being  called  “soft”,  generally 
meaning  that  it  is  expressed  in  language.  This  has  given  rise  to  accelerated  development 
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of  computational  methods  to  process  textual  input,  but  the  state  of  the  art  remains  far 
from  achieving  the  goal  of  natural  language  understanding.  As  a  result,  analysis  methods 
are  further  complicated  by  the  subtleties  and  vagaries  of  language  and  also  the 
complexities  of  semantics  in  the  large. 

We  also  point  out  that  these  problem  types  impute  a  relational  focus  onto  the 
requirements  for  analysis  processes  [3],  This  is  in  part  because  the  inherent  focus  for 
analyses  is  on  situational  and  impact  estimates,  rather  than  the  tactical  type  focus  on 
singular  or  grouped  physical  entities.  One  general  characterization  of  a  “situation”  can  be 
said  to  be  “a  set  of  entities  in  a  set  of  relations,”  and  for  the  types  of  problems  of  interest 
here,  the  extent  of  relation-types  is  extensive  and  of  high  dimension.  Said  otherwise, 
states  of  affairs  are  composed  of  entities  (in  the  most  general  sense  of  the  tenn  - 
including  physical  objects,  properties,  attributes,  mental  events,  temporal  sequences,  and 
the  like)  that  stand  in  relation  to  one  another;  it  can  be  argued  then  that  to  describe  a 
“state  of  affairs”  or  a  “situation”  is  to  describe  a  relational  complex.  Importantly,  these 
relations  for  the  class  of  problems  addressed  here  are  largely  among  people — humans — 
and  thus  the  focus  of  analysis  is  also  about  “human  dynamics”,  and  we  see  for  example 
the  Defense  Science  Board  urging  for  initiatives  to  enable  “Understanding  Human 
Dynamics”  [4]. 

And  now,  roughly  since  some  papers  were  released  by  the  Office  of  the  Under 
Secretary  of  Defense  for  Intelligence  (OUSDI)  in  2010,  we  see  the  tenninology  of 
“Activity-based  Intelligence  (ABI)”  entering  the  discussions  about  intelligence  analysis. 
One  source  [1]  defines  ABI  as  “a  discipline  of  intelligence  where  the  analysis  and 
subsequent  collection  are  focused  on  the  activity  and  transactions  associated  with  an 
entity,  a  population  or  an  area  of  interest.”  ABI  has  also  been  labeled  as  a  new 
“intelligence  tradecraff ’,  and  is  also  closely  coupled  to  the  notion  of  “human  domain 
analytics”  just  mentioned.  The  purpose  of  ABI  has  been  characterized  in  [1]  as  involving 
the  following  five  elements: 

•  Collect,  characterize  and  locate  activities  and  transactions; 

•  Identify  and  locate  actors  and  entities  conducting  the  activities  and  transactions; 

•  Identify  and  locate  networks  of  actors; 
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Understand  the  relationships  between  networks; 
Develop  patterns  of  life. 


Developing  patterns  of  life  (an  ill-defined  tenn)  also  requires  an  understanding  of 
relationships  between  various  entities  and  their  activities  and  transactions.  In  concert 
with  these  elements  are  certain  data-centric  principles,  according  to  Quinn,  in  [5],  who 
cites  these  principles  as: 

•  “geo-reference  to  discover,”  which  means  persistently  collecting  data  on  activity 
and  transactions  over  a  broad  area  or  with  a  variety  of  sources,  then  storing  it  in  a 
database  to  be  discovered  later  when  it  intersects  with  other  data. 

•  “sequence  neutrality,”  which  connects  to  the  idea  of  non-linearity,  and  that  data 
related  to  an  activity  may  not  exhibit  linear  temporal,  spatial,  or  other  types  of 
sequences — this  principle  clearly  relates  to  the  one  above. 

•  “data  neutrality,”  or  the  idea  that  all  data  is  good  and  not  to  be  biased  toward  any 
one  data  source. 

This  article  also  mentions  the  notion  of  knowledge  management,  the  need  for 
which  we  agree  with  strongly,  as  this  function  is  critical  in  any  adaptive  multi-INT 
learning  or  discovery-based  inferencing  or  estimation  process.  These  principles  describe 
ABI  as  “non-linear”  but  we  see  this  as  simply  descriptive  of  an  analysis  method  that  has 
the  characteristics  of  learning  and  discovery  procedures,  being  iterative  and  involving 
forward  and  backward  inferencing  and  verification  operations.  Such  processes  are  also 
described  in  the  latest  characterizations  of  intelligence  analysis  as  comprising 
“sensemaking”,  one  description  of  which  involves  “articulating  and  structuring  the 
unknown”  [6].  Pirolli  and  Card,  in  [7]  provide  what  is  the  most-cited  description  of  the 
sensemaking  process  as  a  basis  of  intelligence  analysis,  involving  iterative  “information 
foraging”  in  a  nonmonotonic  analysis  activity. 

Much  has  been  said  about  ABI  in  the  trade  publications  of  the  intelligence 
community  (for  example,  see  [1,  5,  8,  9,  10])  but  there  has  been  very  little  published  in 
the  scientific  literature  that  describes  particular  and  novel  technical  methods  that  are 
explicitly  supportive  of  and  traceable  to  an  ABI  application.  Our  purpose  in  this  report  is 
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to  offer  a  survey  of  algorithmic-level  literature  we  have  gathered  and  reviewed  that  has 
been  directly  applied  to  ABI  analysis,  as  well  as  offering  thoughts  about  and  descriptions 
of  scientific  techniques  having  functional  relevance  to  ABI  drawn  from  different 
application  areas  that  we  believe  have  potential  utility  for  ABI. 
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II.  ABI  REQUIREMENTS  AND  CHALLENGES 


In  [1 1],  Tse  provides  both  a  good  ABI  overview  and  offers  a  copy  of  an  ABI 
“Hard  Problems”  list  (developed  by  the  US  Geospatial  Intelligence  Foundation)  that  we 
copy  below: 

1 .  Advanced  ABI  data  set  analysis,  including  large  and  small  data  sets  of  single, 
multi-INT,  and  multi-source  types 

2.  Automation  of  capabilities  to  enable  anticipatory  analysis  including  alerts, 
anomaly  detection,  change  detection,  object  identification,  etc. 

3.  Semantic  search,  data  mining,  knowledge  disco  very/analysis,  automated 
learning  processes,  and  advanced  search  techniques 

4.  Responsive,  reliable,  and  efficient  cross-discipline/INT/security  level  content 
storage,  discovery,  retrieval  and  correlation 

5.  Quantitative  handling  and  propagation  of  uncertainty/confidence  across 
variable  fidelity/  pedigree  sources  to  support  decision  making 

6.  Geolocation  and  registration  of  activities  and  transactions  with  defined 
uncertainty  to  support  spatial  and  temporal  correlation  across  multi-INT  data 

7.  Foundation  content  domain  that  supports  multi-source  integration  on  contextual 
information  including  human  geography,  foundation  GEOINT,  and  structured 
observables 

8.  An  integrated  end-to-end  multi-INT  ABI  framework  which  enables  a  flexible 
and  discoverable  data  processing,  exploitation,  and  dissemination  management  enterprise 

9.  Modeling  of  activities,  facilities,  entities  and  patterns-of-life  (including 
complex  activities)  to  enable  anticipatory  analysis 

10.  Rapid  and  relevant  multi-INT  information  correlation  across  disparate  sources 
(including  unformatted/unregulated  and  non-temporal/non-spatial  data) 

1 1 .  Integration  of  non-traditional  sensor  modalities  (ie:  gravitometry,  acoustics, 
MASINT,  other  unique  sources) 

12.  Distributed,  advanced,  and  automated  data  processing  (large  and  small  data 
sets;  single-INT  and  multi-INT) 
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13.  Preparation  of  system  data  for  more  effective  cross-sensing  domain  data 
integration 

14.  Automated  processing  of  activities  and  transactions  including  extraction, 
identification,  tracking,  recognition,  and  filtering 

15.  Automated,  synergistic  tipping  and  cueing  across  multiple  sensors  in 
operationally  relevant  timelines 

16.  Integrated  work  flow  composition  to  enable  responsive  analysis  capabilities 

These  “problems”  are  by  and  large  descriptive  of  processing  and  algorithmic  type 
capability  requirements  but  do  not  offer  any  requirements  in  terms  of  the  needs  of 
operational  problem  analysis — that  is,  the  type  of  “answers”  required — the  inferences  and 
estimates  or  hypotheses  desired.  Apart  from  “patterns  of  life”,  there  is  no  specificity 
regarding  the  output  of  any  given  supportive  technology.  While  the  list  is  helpful,  it  also 
lacks  quantification  in  terms  of  scale  for  the  implied  independent  variables;  notice  that 
the  word  “activity”  is  not  in  the  list.  Interestingly,  in  relation  to  the  “answers”  remark 
above,  the  list  does  not  address  the  need  for  a  taxonomy  or  ontology  of  “activities”.  In 
[12],  a  distinction  is  made  between  “actions”,  defined  as  simple  motion  patterns  typically 
executed  by  a  single  human  or  entity,  and  “activities”  that  involve  coordinated  actions 
among  a  small  number  of  humans  or  entities  (see  the  section  on  Hard  and  Soft 
Infonnation  Fusion  below).  The  purpose  of  this  report  is  not  to  enter  into  a  long  treatise 
on  the  philosophical  and  ontological  aspects  regarding  the  nature  and  definitions  of 
actions,  activity,  activities,  etc.  but  it  would  seem  that  collecting  such  material  should  be 
one  of  the  foundational  elements  of  formalizing  the  nature  of  ABI,  toward  helping  to 
define  what  algorithmic  methods  may  be  helpful  in  activity-based  analyses.  Our  web 
search  on  this  topic  rather  surprisingly  yielded  relatively  few  pertinent  citations  but  we 
offer  a  few  remarks  from  them.  The  edited  text  [13]  has  some  relevant  chapters 
addressing  the  ontological  aspects  of  actions  and  activities;  Chapter  1  of  [13],  by  E.  J. 
Lowe  argues:  “Any  comprehensive  theory  of  action  should  have  something  to  say  about 
the  ontology  of  actions”,  a  point  we  agree  with.  This  interesting  chapter  discusses  the 
idea  of  agents — that  perform  actions — and  also  the  notions  of  intentionality  in  action,  and 
whether  events  and  actions  are  equivalent,  and  also  the  issue  of  causation  of  actions, 
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activities,  or  events.  We  stop  here  on  these  philosophical  subtleties  but  it  should  be  clear 
that  moving  ahead  with  formalization  of  ABI  as  a  discipline  will  require  addressing  these 
and  other  points.  A  possible  complex  activity  taxonomy  is  offered  in  [14]  that  nominates 
six  classes  as  in  Figure  1 : 


Figure  1.  Activity  Structures  Making  up  Complex  Activities  (from  [14]) 

A  few  other  references  are  in  [15,  16]  that  introduce  yet  other  issues  regarding  the 
formal  specification  of  activities  such  as  the  temporal  aspects  of  actions  and  activities. 


All  of  the  points  in  the  above  list  and  discussion  are  helpful  toward  understanding 
the  features  of  ABI  but  our  focus  is  on  the  specification  of  methods,  techniques  and 
especially  algorithmic  and  analytical  capability  requirements  that  the  above  list  does  not 
explicitly  define. 
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III.  A  SEARCH  FOR  TECHNOLOGIES 


In  addition  to  studying  the  IC  popular  literature,  we  conducted  a  literature  search 
on  the  web,  with  an  explicit  focus  on  ferreting  out  any  scientific  works  that  directly 
explore  ABI-related  problems  and  that  nominate  and  hopefully  test  and  evaluate 
prototype  technological  and  algorithmic  methods.  Using  our  best  intuition  and 
experience  in  nominating  adjunct  areas  of  study  to  ABI,  we  also  explored  the  literatures 
of: 

•  Tactical  Intelligence  -a  subset  of  works  on  this  topic  that  focus  on  activity  pattern 
recognition 

•  Political  Unrest  and  Societal  Radicalization  -as  we  judge  that  assessing  precursor 
activities  leading  to  radicalization  could  be  one  analysis  goal  of  ABI 

•  Ambient  Intelligence  -that  area  of  study  that  explores  methods  to  design  smart 
environments  by  fusing  and  exploiting  sensor-laden  environments  toward 
inferring  everyday  life  activities,  tasks  and  rituals — these  are  “activity-based” 
studies  on  human  behaviors  conducted  from  close-range  sensing 

•  Action  and  Behavior  Recognition  Literature  -these  from  serendipitous  searching 
on  these  topics 

•  Hard  and  Soft  Infonnation  Fusion  Technologies  -these  quite-new  methods  that 
are  studying  automated,  algorithmic  ways  to  exploit  both  traditional  sensor  data 
and  the  broad  range  of  data  related  to  ABI  comprised  of  textual,  linguistically- 
framed,  semantically-rich,  and  often  unstructured  data  classes. 


We  point  out  that  apart  from  any  works  directly  related  to  ABI  as  derived  from 
long-range  sensing  (our  central  focus),  any  promising  methods  that  could  be  identified 
from  other  related  application  areas  and  problems  immediately  impute  the  issues  of 
scalability  and  robustness  assessment  of  such  methods.  Scalability,  as  a  property  of 
systems,  is  generally  difficult  to  define  and  in  any  particular  case  it  is  necessary  to  define 
the  specific  requirements  for  scalability  on  those  problem-space  dimensions  which  are 
deemed  important.  These  dimensions  are  the  independent  variables  that  “scale”  the 
problem  space,  and  would  allow  a  judgment  of  similarity  of  problem  spaces  to  be 
quantitatively  assessed.  Scalability  can  be  thought  of  as  involving  a  search  for  invariants 
in  the  different  problem  spaces,  in  the  sense  of  these  meaningful  dimensions.  One 
example  of  such  variables  for  the  class  of  multisensor-multitarget  tracking  (multi-INT) 
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problems  are  the  inter-object  spacing,  object  maneuverability,  and  the  sensor  sampling 
rate  (e.g.,  see  [17,  18]),  that  drive  the  difficulty  of  data  association  and  track  estimation 
for  fusion-based  solutions.  This  idea  is  closely  linked  to  the  need  for  an  ontology  of 
activities  that  can  aid  in  defining  types  of  activity-estimation  problem  classes,  and  the 
associated  variables  that  define  any  class  of  activities. 

A.  SEARCH  RESULTS 

The  search  results  here  are  based  on  a  literature  base  of  over  50  journal-type 
publications,  and  space  does  not  permit  a  discussion  on  all  of  the  findings  and  judgments 
from  those  findings.  We  focus  on  communicating  the  nature  of  the  approaches  from  a 
top-level,  showing  how  these  researchers  think  about  activity-based  problems  and 
analyses,  as  well  as  trying  to  characterize  the  nature  of  specific  algorithmic  solutions, 
blending  top-down  and  bottom-up  views. 

B.  ABI-  AND  TACTICAL  INTELLIGENCE-SPECIFIC  APPROACHES 

Here  we  found  rather  few  works  that  were  distinctive  in  taking,  what  in  our 
judgment  was,  an  ABI-type  focus  we  believe  the  IC  is  interested  in  (we  point  out  that  we 
did  not  search  on  or  include  works  dealing  with  social  network  analysis  (SNA)  and 
related  topics  which,  while  related  to  ABI  purposes,  focus  more  on  such  issues  as 
organizational  structure,  high-value  individuals,  and  notions  of  prestige,  etc.  as  typical  of 
SNA  analyses).  Broadly  speaking,  these  works  did  employ  an  activity-based  approach  to 
identify  adversarial  organizational  structures  and  related  missions,  so  in  a  sense  they  are 
not  far  from  SNA-type  methods  but  do  address  the  behavioral  aspects  more  intently  than 
SNA  techniques.  One  of  these  papers  [19]  does  provide  an  informative  overview  of 
some  approaches  and  specific  methods  for  the  problem  of  modeling  and  learning  of 
behavioral  patterns,  and  we  summarize  these  viewpoints  below. 

We  consider  that  one  fundamental  specific  capability  required  for  addressing  ABI 
problems  is  detection  and  labeling  of  coordinated  behavior,  action,  and  activity  patterns. 
A  non-trivial  aspect  and  characteristic  of  this  problem  is  that  the  patterns  evolve  in  time; 
that  is,  a  pattern  develops  over  and  consumes  time  in  its  evolution,  and  is  thus  only 
“present”  after  some  time  interval  has  occurred.  In  addition,  the  set  of  activities  of 
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interest  will  typically  range  over  layers  or  classes,  and  these  structures  need  to  be  built  up 
from  available  data  (this  again  raises  the  issue  of  ontological  specification  of  the  overall 
class  of  activity  structures  of  interest.)  One  example  of  a  behavioral  hierarchy  is  shown 
in  [19]  as  in  Figure  2: 


I  ] 


1  I 


5  Spatio-tomporal 
Pattoms 


4  Multi-actor 
Functions/Tasks 


3  Activities  of 
Individuals 


nJlji 


Supply  Chain 


Warehouse 


Avr  IT 


fur  chasing 


I  Data  Capture  from  environment 


Building  Group  Labeling 
Plan  recognition 


Building/Area-Function 

Classification 


Activity  Recognition 


Entity  recognition,  object 
tracking,  event  detection 


Digital  data  stream 


Figure  2.  Example  Behavior  Class  Hierarchy  (from  [19]) 


Although  this  paper’s  focus  is  not  specifically  on  the  multisource  information 
fusion,  multi-INT  issues  (although  we  point  out  some  important  fusion  issues  in  our 
remarks),  we  note  that  the  development  of  inferences  and  estimates  of  any  given  type  of 
behavior  or  activity  as  for  example  in  Fig.  2  clearly  will  involve  the  design  and 
specification  of  a  state  of  the  art  information  fusion,  multi-INT-based  process.  Some 
additional  remarks  on  fusion  issues  are  offered  in  the  section  below  when  we  address  the 
“hard  and  soft  infonnation  fusion”  process  and  its  complexities. 


As  [19]  also  points  out,  one  is  not  only  interested  in  observable  aspects  of 
activities  but  also  the  underlying  causal  motivations.  Classification  of  activities  is  also 
related  to  issues  of  resolution  and  related  to  this  is  the  issue  of  “A2AD”  or  anti-access, 
area  denial  aspects  of  any  observational  operation.  (Recent  papers  in  the  defense  and  IC 
community  have  pointedly  remarked  that  virtually  all  observational  data  in  Iraq  and 
Afghanistan  have  been  taken  from  nadir  viewpoints  and  at  close  range;  other  modern 
problems  such  as  ABI  are  not  likely  to  enjoy  such  favorable  conditions.)  Additional 
complexities  in  behavioral  classification  defined  in  [19]  are  the  influence  of  contextual 
factors  (time-of-day,  weather,  etc.),  latent/unknown  factors,  and  of  course  the  errors, 
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uncertainties,  and  ambiguities  in  observational  data.  Specific  methods  cited  for  activity 
recognition  in  [19]  are:  n-grams,  stochastic  context-free  grammars  that  provide  syntactic 
recognition  of  the  activity  process’  structure,  Hidden  Markov  Models  that  enable  both 
recognition  of  the  temporal  event  structure  of  the  activities  from  partially  observable 
phenomena  and  support  unsupervised  learning-based  estimation  of  activities.  ABI 
analyses  will  not  only  be  interested  in  the  activities  per  se  but  what  underlying  cause  or 
function  is  occurring;  [19]  addresses  this  issue  in  various  specific  techniques  for  function 
recognition,  such  as:  Probabilistic  Latent  Semantic  Analysis  and  the  related  Latent 
Dirichlet  Analysis  technique,  and  the  class  of  network  or  graph  pattern  matching 
methods.  One  concept  for  an  architecture  that  puts  all  this  together  is  shown  in  [19]  as  in 
Figure  3: 


Figure  3.  Notional  “Geo-Spatial  Behavior  Pattern  Analysis  System”  (from  [19]) 


Levchuk  et  al  [19]  conclude  their  paper  with  comments  regarding  research  needs 
toward  developing  the  types  of  capabilities  considered  to  be  required  for  adversarial 
behavior  recognition.  These  are  shown  below,  with  our  own  comments  added: 


•  Vocabulary  of  functions,  activities,  and  events:  said  otherwise  this  is  the  need  for 
behavior,  action,  activity,  function  ontologies 

•  Choice  models:  these  are  indirectly  intent  or  causal  type  models 

•  Development  of  models  of  activities,  functions,  and  mission  patterns:  at  least  for 
model-based  approaches,  models  of  course  are  needed,  but  learning,  discovery 
and  inductively  based  methods  are  also  needed 

•  Tracking  the  state  of  the  behavior  signature  and  iterative  prediction  refinement: 
indirectly  this  is  a  remark  that  much  current  work  has  been  batch/forensically- 
motivated;  this  is  a  statement  of  need  for  streaming  and  also  predictive/recursive 
methods  (see  the  section  below  regarding  hard  and  soft  infonnation  fusion) 
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•  Understanding  how  the  behaviors  change  and  can  be  influenced  over  time:  more 
or  less  an  aspect  of  the  above  in  that  behavioral  state  change  models  are  needed; 
again  this  could  be  considered  as  part  of  the  ontology  specification,  as  many 
ontologies  incorporate  temporal  aspects 

•  Dealing  with  data  at  different  levels  of  time  and  infonnation  hierarchy:  in  our 
view,  this  is  essentially  the  need  for  well-designed  fusion  systems 

•  Integration  of  the  reasoning  and  collection  planning:  this  is  fundamentally  a  need 
for  an  adaptive,  closed-loop  process.  This  involves  however  the  issue  of  “design 
authority”  meaning  whether  the  reasoning  (eg  fusion)  process  designer  has 
authoritative  control  over  collection  or  not  (sometimes  problematical  in  the  IC 
environment);  if  yes,  then  an  adaptive  process  can  be  designed — if  not,  these 
processes  are  disconnected 

•  Reasoning  about  the  behaviors  as  well  as  actors  conducting  those  behaviors,  and 
corresponding  data  association  problems:  again  we  see  this  as  a  part  of  fusion 
process  design,  as  association  is  a  fundamental  function  of  fusion  process  design 

•  Testing  and  training:  these  aspects  are  fundamental  to  any  well-developed 
technological  capability,  and  especially  important  are  the  fonnation  of  associable 
multisource  data  sets  for  development,  testing  and  evaluation 


In  the  areas  of  ABI  and  what  we  informally  label  as  tactical  intelligence,  we 
survey  some  dozen  papers  in  Table  1  below  that  we  believe  are  reflective  of  our  overall 
survey;  these  are  just  a  sample.  The  table  columns  include  the  citation,  the  nature  of  the 
input  data  dealt  with,  the  operational  problem  addressed,  the  technical  methodology 
studied,  and  remarks  about  any  results  presented.  There  is  no  prioritization  or  ranking 
implied  in  the  table. 


Table  1.  Sample  Survey  Results  for  Papers  on  ABI  or  Tactical  Intelligence 


Paper/Citation 
(see  References) 

Observational 

Data 

Operational 

Problem 

■Hi 

Results 

[20]  Recognition 
of  Coordinated 
Adversarial 
Behaviors  from 
Multi-Source 
Infonnation 

Intell  Rpts  (No 
association 
fonnalism)  re 
Transactions; 

some 

clustering  of 
reports 

Adversarial 
organizational 
understanding 
,  mission 
estimation 

Model¬ 
matching 
(attributed 
graphs)  using 
graduated 
assignment 
algorithm 

Simulation-based 
results;  70%  type 
accuracy 

[21]  Identifying 
the  Enemy  -  Part 

I:  Automated 

Intell  Rpts  and 
Intercepts; 
association  is 

Adversarial 

organizational 

understanding 

Probabilistic 

attributed 

Graph 

Simulation-based 
results;  high 
accuracy 
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Network 

Identification 

Model 

Individual 
data-to-model, 
not  data-to- 
data 

Do  not  deal 
with  entity 
extraction 
issues, 

imperfections 

,  mission 
estimation 

[22] 

Supply-chain 

Adversarial 

Identification  of 

reports 

facility 

Adversarial 
Activities: 
Profiling  Latent 
Uses  of  Facilities 
from  Structural 
Data  and  Real¬ 
time  Intelligence 

identification 

[23]  Activity 

Set  of  distinct 

Broad  area 

Recognition  in 

tracklets  of 

kinematically- 

Wide  Aerial 

reasonable 

based  activity 

Video 

duration. 

labeling 

Surveillance 

presumably 

Using 

derived  from 

Entity 

wide-area 

Relationship 

surveillance 

Models 

imagery;  real 
dataset 

[24]  Relations  as 

Canadian  AI 

Estimates  of 

Context  to 

synthetic  data; 

suspicious 

Improve  Multi- 

reports  of 

rendezvous 

Target  Tracking 

maritime 

among 

and  Activity 

tracks  and 

various  ships 

Recognition 

suspected 

interplatform 

relations 

[14]  Recognizing 

UGS-like 

Notional 

Activity 

GROUND 

pattem-of-life 

Structures  in 

sensor  network 

activities  on 

Massive 

the  ground 

matching 

algorithm. 


Probabilistic 

attributed- 

structural 

pattern 

consistency 


Good  performance 
if  key  attributes 
contained  in  input 
reports 


Entity- 

Relation 

Models  for 

various 

activities; 

existing/com 

mercial 

RDBMS 

Search  tools 


Relational 

Particle  Filter 

that  projects 

future  entity 

tracks 

(includes 

relational 

transition 

estimates); 

Relational 

Dynamic 

Bayesian 

Networks 


(1)  A 

methodology 
is  presented 
for 


Good  performance 
for  simple 
maneuvers/behavi 


Limited 

experiments;  fair 
accuracy 


Concept  paper;  no 
empirical  results. 
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Numbers  of 
Simple  Events 
Over  Large 
Areas 


describing 

the 

“structure”  of 
complex 
activities  that 
span  large 
space-time 
volumes;  this 
is  a  kind  of 
ontology. 
(2)Examples 
of  several 
techniques 
are  presented 
for 

recognizing 

complex 

activity 

patterns 

given  large 

numbers  of 

simple  events 

from  a 

ground 

sensor 

network 


Nominates 
fusion-based 
methods  and 
Transferable- 
Belief  Model 
and  Analysis 
of 

Competing 

Hypotheses 

techniques. 


Interframe 

compression, 

difference 

coding, 

motion 

compensatio 

n,  techniques 

from 

biosurveillan 


Technology 
planning  document 


Concept  paper;  no 
empirical  results. 
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[27]  Adversarial 

Knowledge 

Discovery 

Records  of 
various  type, 
mixed  with 
“normal” 
records 

Broad  class  of 
problems 
where 
adversaries 
are  outliers 

Distribution, 

Density,  and 

Hierarchicall 

y-based 

Clustering; 

Graph 

partitioning 

schemes; 

ensemble  and 

random 

forest 

predictors 

Concept  paper;  no 
empirical  results. 

[28]  CMAP:  a 
Flexible  and 
Efficient 
Framework  for 
Constraint-based 
Mining  of 

Activity  Patterns 

Records  of 
various  type 
that  can 
structured  into 
logical  clauses 

Use  Case 
involving 
intell  on 
aircraft 
operations 

Constraint- 
based  Data 
mining 
approach  to 
discover 
patterns  as 
logic  clauses 
(entities 
connected  by 
relational 
sequences) 

Good  results  on 

Use  Case; 
generalization 
under  study — 
trades  off  number 
of  constraints  vs 
precision,  recall 

[29]  Group 
Activity  Analysis 
for  Persistent 
Surveillance 

Streaming 

video 

Insurgent, 

terrorist 

activities; 

focus  on 

group 

activities; 

Game-based 

simulation 

expts 

Model-based 

approach 

using  Event 

graphs, 

Probabilistic 

Petri  nets; 

exploits 

ontological 

infonnation 

Good  Precision, 
Low  Recall  on  Use 
Case 

[30]  Automated 
Learning  and 
Visualization  of 
Traffic 

Patterns  from 
Multi-INT  Fused 
Tracks  (DARPA 
INSIGHT  Pgm) 

Multi-INT 

Insurgent 
activities; 
expts  at  the 

Natl  Training 
Ctr;  forensic 
approach 

Hierarchical, 
data  driven 
learning 
approach  that 
builds  on 
fused  tracks, 
cluster 

development, 

edge 

bundling 

Overall  good 
performance  for 
NTC  Use  Cases; 
provides  traffic 
pattern  attributes 
but  does  not  assign 
semantic  activity 
labels 
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Some  of  these  papers  address  the  broad  issues  in  designing  approaches  to  ABI- 
type  problems,  such  as  [14,  25,  26,  27].  Of  the  papers  that  experiment  with  specific 
techniques,  the  techniques  span  a  variety  of  probabilistic  pattern-matching  approaches, 
with  many  using  graph-based  strategies.  Notions  of  inter-source  data  association  are 
surprisingly  missing  in  many  of  the  references.  It  is  clear  that  robust  toolsets  for  ABI 
will  require  layered  and  hierarchical  approaches  that  span  localized,  single-entity 
behaviors  to  abilities  to  semantically  label  group  or  otherwise  aggregated  activities.  Our 
list  of  named  methodologies  derived  from  these  and  related  readings  are: 

•  Probabilistic  plan  recognition 

•  Probabilistic  attributed  graph  matching 

•  Probabilistic  attributed-structural  pattern  consistency 

•  Relational  Dynamic  Bayesian  Networks 

•  Assignment  algorithms 

•  Hidden  Markov  methodological  variants 

•  Entity-Relation  Models 

•  Relational  Particle  Filter 

•  Constraint-based  pattern  mining 

As  might  be  expected,  many  of  the  methods  studied  are  based  on  the  class  of 
graph-analytic  methods,  since  activities  are  patterns  and  a  very  efficient  way  to  model 
and  represent  patterns  is  through  the  use  of  graphical  representations.  There  are  some 
interesting  state  of  the  art,  survey  type  papers  on  graphical  methods  in  [3 1,  32],  Figure  4, 
from  [31],  shows  a  taxonomy  of  graphical  representations  classified  according  to  their 
temporal  dependence  and  structural  nature. 
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Figure  4.  One  Classification  of  Graph-Analytic  Representations  and  Methods  (from 

[31]) 

We  emphasize  however  that  providing  effective  technological  and  automated 
support  to  ABI-type  analyses  will  not  be  fully  aided  or  solved  by  singular  methods  but  by 
a  layered,  multi-tool  architecture;  it  is  because  of  this  that  we  have  included  some  of  the 
“concept”  papers  in  our  review,  since  developing  strategies  for  toolset  architecture  will 
be  as  important  as  designing  and  selecting  the  component,  individual  tools  (as  depicted  in 
Figs  2  and  3). 


C.  APPROACHES  FOR  ASSESSING  STAGES  OF  POLITICAL  UNREST 
AND  SOCIETAL  RADICALIZATION 


The  importance  of  understanding  population  dynamics  has  been  well 
demonstrated  in  the  Iraq  and  Afghanistan  experiences  over  the  last  decade.  As  the 
United  States  approaches  the  post-Afghanistan  era,  and  likely  enters  into  an  era  where  the 
primary  missions  will  involve  stability,  security,  transition,  and  reconstruction  efforts, 
continued  development  of  capabilities  to  aid  in  understanding  important  societal  trends 
will  remain  a  technological  priority.  This  need  was  recognized  as  early  as  in  2009,  when 
the  Defense  Science  Board  issued  its  report  on  “Understanding  Human  Dynamics”,  as 
previously  pointed  out  [4],  that  reviewed  and  evaluated  the  state  of  the  art  at  that  time.  In 
searching  for  ABI-related  literature,  we  encountered  a  set  of  materials  on  political 
dynamics  that  we  felt  to  be  relevant  to  ABI  analytical  methods,  and  summarize  those 
findings  in  this  section. 
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In  these  papers,  there  is  a  direct  focus  on  what  could  be  called  “patterns  of  life”- 
based  analyses,  derived  from  an  integrated  framework  of  pattern-discovery  type  tools;  all 
of  them  take  what  can  broadly  be  called  a  computational  social  science  approach.  In  [33] 
for  example,  a  suite  of  Social  Network  Analysis,  Sentiment  Analysis,  and  Investigative 
Data  Mining  tools,  linked  to  various  text-processing  components  (since  some  of  the  data 
in  this  case  are  Open  Source  and  largely  textual)  such  as  Named  Entity  Recognizers  and 
Relation  Extractors  are  tied  in  to  various  “INT”  feeds  (although  this  paper  again  does  not 
address  the  Data  Association  problems).  In  a  similar  way,  [34]  develops  a  framework  for 
analysis  of  trends  in  diaspora  type  environments  that  employs  pattern-matching  methods 
in  concert  with  a  theory  called  Social  Movement  Theory  [35]  that  examines  resource 
mobilization,  opportunities  and  constraints,  and  stochastic  situational  frame  structures  to 
develop  a  sense  of  what  is  called  the  diffusion  of  social  movement  across  different  social 
strata,  as  an  indicator  of  how  significant  social  movements  propagate.  It  can  be  seen 
from  the  table  entry  that  yet  other  tools  are  used  in  a  multitool  approach.  In  [36],  a  kind 
of  set-covering  approach  is  used  to  address  the  challenge  of  finding  the  smallest  (or  at 
least  a  small)  set  of  patterns  which  “cover”  all  observed  occurrences  of  an  event  of 
interest,  using  pattern  analysis  and  backward  chaining.  As  can  be  said  as  true  of  sensibly 
all  ABI-related  methods  surveyed  here,  [37]  argues  that  there  is  no  single  major  indicator 
of  activities  or  phenomenology  of  interest  in  these  problems;  all  analyses  will  require  a 
set  of  heterogeneous  tools,  and  an  imputed  challenge  will  be  in  developing  automated 
support  toward  integrating  tool  outputs  into  a  coherent  whole  hypothesis  of  interest. 


Table  2.  Sample  Survey  Results  for  Papers  on  Political  Unrest  and  Societal 
Radicalization 


Paper/Citation 

(see 

References) 

Observational 

Data 

Operational 

Problem 

Technical 

Methodology 

Results 

(33)  Detecting 
Social 
Polarization 
and 

Radicalization 

Open-source 
data  plus 

Intell,  Multi- 
INT 

Detect  trends 
and  assess 
risks  of  social 
polarization 
and  violent 
radicalization 

Named  Entity 

Recognizer 

Relation 

Extraction 

System 

Sentiment 

Analysis 

System 

None;  proposed 
prototype 
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(34) 

Transnational 

Islamic 

Activism  and 

Radicalization: 

Patterns, 

Trends,  and 

Prognosticators 


None; 

framework 

paper 


Theoretical 

and 

conceptual 

framework 

for 

understanding, 

recognizing, 

and 

anticipating 
Islamic  social 
refonn 
movements 


SNA 

Investigative 
Data  Mining 


Social 

Movement 

Theory 

Linear 

Temporal 

Logic 

Influence 

Diagrams 

Various  other 

theories  and 

models 


Provides  a  causal 
structure  for  the 
interrelationships 
among  the  myriad 
features  of  a  social 
movement. 


(36) 

Computational 
Methods  to 
Discover  Sets 
of  Patterns  of 
Behaviors  that 
Precede 

Political  Events 
of  Interest 


(37)  The 
Violent  Islamic 
Radicalization 
Process:  A 
Framework  for 
Understanding 


High  level 
political 
factors  drawn 
from  separate 
socio-political 
analyses 


None; 

framework 

paper 


An  approach 
to  identify  sets 
of  patterns  of 
behaviors 
which  precede 
political 
events  of 
interest  such 
as  the  onset  of 
regime 
change, 
insurgency, 
ethnic 

violence,  etc. 


Building  of  a 
framework  to 
understand 
social 

polarization 
and  violent 
radicalization 


Greedy  set 

covering 

pattern 

analysis  and 

backward 

chaining 


No  empirical  results 


The  findings  herein 
confirm  wide 
agreement  that  no 
single  prescribed 
path  of 

radicalization  exists. 
This  study  has  also 
identified  the 
criticality  of  the 
mobilization 
component  of  the 
radicalization 
process.  It  has 
identified  behaviors 
and  combinations  of 
behaviors  to  be 
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studied  further  as 
indicators  of 
radicalization  and 
mobilization  leading 
to  acts  of  terrorism. 

Our  list  of  named  methodologies  derived  from  these  and  related  readings  are: 

•  Named  Entity  Recognition 

•  Relation  Extraction  System 

•  Sentiment  Analysis  System 

•  Social  Network  Analysis 

•  Investigative  Data  Mining 

•  Inductive  reasoning 

•  Social  Movement  Theory 

•  Linear  Temporal  Logic 

•  Influence  Diagrams 

•  Greedy  set  covering  pattern  analysis  and  backward  chaining 

where,  as  noted  above,  it  is  again  emphasized  that  no  single  approach  or  analytical  tool 
will  be  sufficient  for  a  complete  analysis.  This  list  adds  some  methods  needed  for 
dealing  with  textual  and  linguistic  type  data,  addressing  more  pointedly  the  issues  that 
such  soft  data  introduce — more  is  said  on  these  issues  the  section  below  where  we  remark 
on  the  soft-hard  data  fusion  issue. 

D.  APPROACHES  EXPLORED  IN  THE  FIELD  OF  AMBIENT 
INTELLIGENCE 

Ambient  Intelligence  (AI)  is  a  field  of  study  generally  having  to  do  with  smart 
environments  that  adapt  to  human  activities  within  them.  The  activities  of  interest  are 
usually  labeled  as  “activities  of  daily  life”  (the  abbreviation  “ADL”  is  littered  throughout 
this  literature),  and  an  aspect  of  the  smart  environments  is  that  they  are  instrumented  with 
“multi-INT”  sensors,  whose  data  are  used  for  inferencing  and  estimating  the  type  of 
activities  being  perfonned.  While  this  type  of  application,  involving  close-range  sensing 
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and  no  overt  adversarial  actions  such  as  deception  etc.,  is  not  that  close  to  intelligence- 
based  ABI  applications,  we  include  it  because  this  field  is  quite  active  and  considerable 
levels  of  technology  that  may  be  extensible  to  ABI  are  being  developed.  Although  the 
problem  spaces  are  not  adversarial,  they  are  non-cooperative  and  do  not  depend  on  any 
helpful  actions  on  the  part  of  the  humans  involved. 

Two  of  the  papers  we  include  in  our  survey  (see  the  first  two  entries  in  Table  3) 
are  themselves  surveys  and  so  collectively  provide  a  comprehensive  look  at  the  field, 
although  they  are  a  few  years  old.  In  [38],  a  figure  is  provided  that  neatly  summarizes 
their  view  of  the  AI  field;  that  figure  is  shown  here  as  Figure  5: 


Figure  5.  Overview  of  the  Approaches  Employed  in  AI  (from  [38]) 


Notice  that  [38],  as  we  remarked  earlier,  partitions  the  domain  into  “Actions”  and 
“Activities”,  where  actions  are  characterized  by  simple  motion  patterns  typically 
executed  by  a  single  human  in  these  environments,  and  activities  are  considered  as  more 
complex  and  involve  coordinated  actions  among  a  small  number  of  humans  in  these 
definitions.  Notice  too  that  some  of  the  methods  listed  in  Fig  5  are  the  same  as  or  similar 
to  methods  enumerated  for  the  prior  application  areas,  hinting  at  possible  employment  for 
ABI-type  applications,  if  scalability  issues  can  be  addressed.  The  second  paper  in  Table 
3  [39],  addresses  the  ontology  issue  more  directly,  and  describes  activities  in  the  fashion 
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of  Figure  6,  leading  to  the  formulation  of  a  formal  ontology  briefly  described  in  the 
paper: 


Activity 


Time 

Actor 

superActivity 

Conditions 

atTime 

performedBy 

Location 

V 

dependentOn 

atLo  cation 

O 

typeOf 

r  i 

require 

/  A 

Activity 

Activity 

ID.  Name 
Decryption 

Resources 

resultedln 

has  Entities 

o  o 


0.0 

Environment 

achieve  v  lastFor 

Entities 

Goal 

subActivity 

Duration 

Activity 

Figure  6.  Conceptual  Model  of  an  Activity  (from  [39]) 

As  noted  in  our  introductory  remarks,  we  are  supportive  of  the  need  for  an 
equivalent  specification  of  ontologies  for  any  given  ABI-class  of  problem.  Our  sampling 
of  AI  papers  is  shown  in  Table  3: 


Table  3.  Sample  Survey  Results  for  Papers  on  Ambient  Intelligence 


Paper/Citation 

(see 

References) 

Observational 

Data 

Operational 

Problem 

Technical 

Methodology 

Results 

(38  4)  Machine 
Recognition  of 
Fluman 
Activities:  A 
Survey 

Input  video  or 
sequence  of 
images; 

Activities  of 
Daily  Living 
(ADL); 
Behavioral 
Biometrics 

This  survey 
focuses 
exclusively 
on 

approaches 

for 

recognition 
of  action 
and  activities 
from  video 

Review  a  range  of 
approaches, 
summarizing  benefits 
and  disadvantages; 
summarized  in  figure 
below 

(39  1) 

Ontology- 
based  Activity 

No  explicit 
data;  this  is  a 
survey  paper 

Activities  of 
Daily  Living 
(ADL) 

Description 
logic  based 
semantic 

There  is  a  need  for  an 
explicit  commonly 
agreed  representation 
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Recognition  in 
Intelligent 
Pervasive 
Environments 


(40  2)  Activity  Use  Cases  Activities  of 

recognition  on  employed  Daily  Living 

streaming  dense  (ADL) 

sensor  data  multispectral 


sensors 


(413) 

Exploring 

semantics  in 

activity 

recognition 

using  context 

lattices 


Two  real-  Activities  of 

world  test  data  Daily  Living 
sets  from  high-  (ADL) 
tech  smart 
homes  with 
dense  sensors 


reasoning 
coupled  to 
ADL 

ontologies 

of  activity  definitions, 
i.e.,  ontologies,  for 
activities  that  are 
independent  of 
algorithmic  choices, 
thus  facilitating 
portability, 
interoperability  and 
reuse  and  sharing  of 
both  underlying 
technologies  and 
systems. 

Support 

Vector 
Machine  to 
classify 
activities 

Classifications  of 
activities  ranged  from 
60%  to  75%  accuracy. 
Raises  the  important 
issue  of  interleaved 
events  and  activities. 

An  important 
challenge  with  this 
approach  is  identifying 
the  optimal  window 
size. 

Context 

Lattice 

Outperfonned  Bayes 
and  Decision  Trees  for 
activity  recognition 

IV.  PREVAILING  ISSUES 


A.  THE  ISSUE  OF  FUSING  HARD  AND  SOFT  INFORMATION 

The  experiences  of  Iraq  and  Afghanistan  have  also  shown  that  inputs  and 
supporting  information  (such  as  contextual  infonnation,  either  real-time  or  in  static 
databases)  of  a  textual,  linguistic  type  have  proven  helpful  if  not  critical  toward  yielding 
improved  intelligence  analysis  capabilities.  This  type  of  infonnation  has  come  to  be 
called  “soft”,  to  distinguish  it  from  “hard,  physics-based”  data  provided  by  the  usual 
repertoire  of  ISR  sensing  devices  across  the  electromagnetic  spectrum.  In  most  of  the 
papers  reviewed  here,  such  data  are  considered  to  be  part  of  the  input  or  observational 
data,  along  with  sensor  data;  the  multi-INT  problem  of  interest  can  then  be  characterized 
as  a  hard  +  soft  information  fusion  problem.  However,  very  few  of  the  papers  included 
here  address  the  daunting  challenges  of  the  hard  +  soft  fusion  problem  with  any  rigor. 
Linguistic  data  immediately  impute  the  long-studied  problem  of  natural  language 
understanding,  with  all  of  its  complexities,  as  well  as  introducing  the  complexities  of 
semantics.  It  is  only  in  the  last  5  years  that  the  infonnation  fusion  community  has 
addressed  this  class  of  problem,  and  the  evolving  technological  capabilities  are  only 
recently  beginning  to  evolve  out  of  the  academic  community.  The  Center  for 
Multisource  Infonnation  Fusion  (CMIF)  at  the  University  at  Buffalo,  Harvard  University, 
and  the  University  of  Washington  are  leading  multi-university  teams  in  5-year  programs 
under  Anny  Research  Office  funding  to  study  and  develop  prototype  capabilities  to 
rigorously  deal  with  the  challenges  of  the  hard  +  soft  fusion  problem  (see  [42,  43,  44]  for 
some  overview  papers).  Robust  architectures  that  provide  meaningful  automated  support 
for  ABI  analyses  and  the  evolving  ABI  tradecraft  will  need  to  include  not  only  the  tools 
described  herein  as  derived  from  directly  and  indirectly-related  applications,  but  also 
integrate  the  fonnalisms  of  methods  to  address  the  hard  +  soft  multi-INT  fusion  problem. 
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B. 


EXTENSIBILITY  AND  SCALABILITY 


Finally,  we  offer  a  brief  reminder  that  while  any  given  (and  properly-defined) 
ABI  problem  may  possibly  be  addressed  by  the  various  methods  described  herein,  there 
are  issues  to  consider.  Even  methods  that  have  been  applied  to  what  is  considered  a 
directly-relevant  ABI  problem  need  to  be  carefully  reexamined  for  extension  and 
scalability  to  a  particular  ABI  problem;  while  there  may  be  low-hanging  fruit  across  the 
capability-spaces  reviewed  here,  careful  assessments  of  extensibility  and  scalability  need 
to  be  made  for  future  specific  applications. 
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V.  CONCLUSIONS 


This  report  tries  to  provide  an  exploration  of  the  range  of  current  technological 
methods  being  used  for  ABI  and  ABI-related  problems,  toward  providing  an  initial  view 
of  that  inventory  as  a  resource  for  needed  initiatives  in  well-planned  development  and 
realization  of  the  scientific  foundations  that  will  allow  ABI  to  mature  into  a  new  and 
needed  intelligence  discipline.  Raetz  [45],  offers  the  following  perspective  on  ABI:  “ABI 
holds  the  same  promise  for  imaging  the  Human  Domain  that  the  invention  of  the  camera 
held  for  imaging  physical  terrain;  however,  new  methodologies,  workflows  and 
perspectives  are  needed  to  ensure  that  the  benefits  of  Human  Domain  analysis  are  fully 
explored.”  Raetz  champions  a  layered-graph  system  type  approach  toward  uncovering 
layered  systems  of  what  we  have  called  here  “relational  complexes”  among  sets  of 
activities  and  transactions.  That  approach  at  least  recognizes  that  a  layered,  multi-tool 
type  analytical  framework  is  required  to  address  the  analysis  challenges  of  ABI.  But  the 
integrated  challenges  of  ABI  will  require  defining  and  prototyping  new  analysis  process 
designs  that  effectively  deal  with  big-data  levels  of  multi-INT  that  likely  include  more 
soft  data  than  hard,  equally  large  amounts  of  contextual  data,  suites  of  helpful  but 
disparate  automated  tools,  in  a  discovery/learning/sensemaking  process  that  importantly 
includes  dependencies  on  human  intelligence,  and  finally  is  dynamic  and  adaptive.  Raetz 
is  correct  we  think  in  pointing  out  that  to  achieve  efficient  and  effective  frameworks  for 
ABI  analysis  will  require  new  methodologies  and  workflows  but  we  would  add  that  the 
meta-issues  that  will  throttle  the  ability  to  achieve  such  capability  are  the  identification 
and  enablement  of  a  structured  basic  research  program  to  assess  and  identify  potentially 
new,  innovative,  currently  unknown  foundational  technologies  to  aid  in  analysis,  and  a 
parallel  initiative  to  more  fully  understand  how  to  design  analysis  environments  that 
robustly  support  dynamic  human-machine -based  inferencing,  estimation,  and  reasoning. 
The  Center  for  Multi-INT  Studies  (CMIS)  at  the  Naval  Postgraduate  School  has 
conducted  various  multi-INT  studies  pertinent  to  ABI,  and  is  supporting  the  sponsor 
community  in  providing  national  Multi-INT  research  leadership,  expanding  the  number, 
breadth,  and  depth  of  researchers  conducting  high-value  Multi-INT  research;  these  efforts 
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are  delivering  high-value  research  outcomes,  e.g.,  [46-47].  CMIF  and  CMIS  seek  to 
encourage  similar  emphasis  and  coordination  in  this  area.  We  consider  that  the  role  of 
academia  in  this  overarching  initiative  is  quite  large.  Tse,  in  [1 1],  addresses  this  directly, 
suggesting  “where  academic  partners  can  help”  in  a  list  including:  Automated  policy- 
based  ABI  enterprise  collaboration  methods  and  tools,  Multi-modality  information 
representation,  organization,  access  and  mining,  Information  integration  and  multi¬ 
modality  intelligence  association  (as  we  have  pointed  out  above,  an  often-overlooked 
requirement),  and  in  Enterprise  Mission  Management,  related  to  our  point  on  the  design 
of  holistic  environments.  As  regards  the  Intelligence  Community,  such  academic 
initiatives  can  be  supported  in  part  by  such  programs  as  the  IC  Postdoctoral  Research 
Fellowship  Program,  if  a  coordinated  focus  on  the  ABI  requirements  can  be  developed. 
In  sum,  there  is  a  need  for  a  partnered,  government-industry-academic  balanced  and 
integrated  approach  to  identify  and  develop  the  scientific  and  technical  foundations  that 
will  support  robust  ABI  analytical  capabilities  going  forward. 
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